Make language names unique #149
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This seems good to me. "Baka (South Sudan/Congo)" should probably be with "DRC" or "Congo-Kinshasa" as "Congo" is ambiguous. Should multiple countries be in a specific order? |
rsheeter
reviewed
May 14, 2024
| names = Counter([lang.name for lang in LANGUAGES.values()]) | ||
| if any(count > 1 for count in names.values()): | ||
| duplicates = {name: count for name, count in names.items() if count > 1} | ||
| pytest.fail(f"Duplicate language names: {duplicates}") |
tests/test_data_languages.py
Outdated
|
|
||
|
|
||
| def test_language_uniqueness(): | ||
| names = Counter([lang.name for lang in LANGUAGES.values()]) |
There was a problem hiding this comment.
When present, I think you want to use lang.preferredName instead of lang.name.
That said, you might consider using that preferredName field in some of the language proto files in this PR. I don't actually see specific examples where that would make more sense, but worth considering.
There was a problem hiding this comment.
Oh boy, that found extra duplicates. :-)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This improves CI testing for textproto parseability, and the uniqueness of language names. I've noticed that we have a system for disambiguating language names: "
<Region>ian <Language>" where there are regional variants, and "<Language>, <Script>" where the language is written in multiple scripts; so I've tried to extend this to the cases which are currently doubling up. Additionally for Baka there are two different, unrelated languages with the same name, so I have disambiguated them with "Baka (<Region>)".I've also removed
mlt_Latn(Maltese) because we have a bettermt_Latn. @moyogo, does that sound fair?